Append data load using SSIS
Hi...I have a task where I need to transform the data from source to destination. The source table has about 40 million rows and keep growing rapidly. At present I am truncating the destination table everyday and reloading the data every day which is a time
consuming process.
I am not able to find any logic to implement the append logic so that I only tranfer the data from last time stamp when the job runs. The source table has four columns with no unique key identifier(all varchar) and and timestamp column. Since this is
a power metering data, the timestamp is the actual meter reading time. The data in the source is coming from the various time zones.
I can not use the timestamp value as the unique identifier...for example if I ran the job today at 12 am, my destination table had the max timestamp row as 12.15 AM.
The next day when I run the job I can not use the condition to get the data from 12.15 onwards from source since there might have some data in the source of before 12.15 am which is not in the destination yet (from the timezones where the time
is lagging).
Please let me know if there is an ulternate way available for this?
Thanks, Gaurav
November 16th, 2010 1:03pm
Sounds like you're screwed.
Any chance of modifying the source table? Add a [DateTime Appended] field with a default value of GETDATE() ?Todd C - MSCTS SQL Server 2005 - Please mark posts as answered where appropriate.
Free Windows Admin Tool Kit Click here and download it now
November 16th, 2010 1:40pm